User interfaces to configure a thermal imaging system

ABSTRACT

A thermal imaging system including at least one thermal imaging device, a server, and at least one mobile device. The thermal imaging device captures thermal images of an environment. The server applies computer vision techniques to the thermal images, detects events of a predetermined type, and generates notifications of the events of predetermined types detected from the thermal images. The mobile device runs a mobile application that is configured to receive the notifications, present user interfaces, receive user annotations of the notifications in the user interfaces, and transmit the annotations to the server. According to the annotations, the server adjusts parameters used in the application of the computer vision techniques and in the generation of the notifications.

RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 16/290,367, filed Mar. 1, 2019, which is acontinuation application of U.S. patent application Ser. No. 16/042,045,filed Jul. 23, 2018, issued as U.S. Pat. No. 10,225,492 on Mar. 5, 2019,both entitled “User Interfaces to Configure a Thermal Imaging System,”the entire disclosures of which applications are hereby incorporatedherein by reference.

The present application relates to U.S. patent application Ser. No.15/607,345, filed May 26, 2017, and U.S. patent application Ser. Nos.15/797,693 and 15/797,999, both filed Oct. 30, 2017, the entiredisclosures of which applications are hereby incorporated herein byreference.

FIELD OF THE TECHNOLOGY

At least some embodiments disclosed herein relate to a thermal imagingsystem in general and more particularly but not limited to userinterfaces for the configuration of the thermal imaging system to detectpredetermined types of events.

BACKGROUND

Thermal imaging can be used for human detection, due to a high contrastof the elevated human body temperature compared to the temperatures of atypical indoor environment. Thermal imaging with low resolution can beused for detecting humans reliably within a typical room area. Theadvantages of low resolution thermal imaging compared to conventionalvideo monitoring using lights visible to human eyes include not only thegood human-background contrast, but also non-intrusion into privacy.When the resolution of thermal imaging is low such that a few pixels areused to represent a person, the thermal image of the person appears as ablob without fine features about the person. Thus, the thermal imagecannot be used to specifically identify the individual. Such technologycan be used for the monitoring of elders and patients to provide care inresponse to certain situations, such as fall, without privacy intrusion.

For example, U.S. Pat. App. Pub. No. 2015/0377711, entitled “Apparatusand Method for Electromagnetic Radiation Sensing”, discloses anapparatus for thermal imaging based on infrared (IR) radiation. Such anapparatus can be used for human detection, fire detection, gasdetection, temperature measurements, environmental monitoring, energysaving, behavior analysis, surveillance, information gathering and forhuman-machine interfaces. Such an apparatus and/or other similarapparatuses can be used in embodiments of inventions disclosed in thepresent application. The entire disclosure of U.S. Pat. App. Pub. No.2015/0377711 is hereby incorporated herein by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings in which like referencesindicate similar elements.

FIG. 1 shows a thermal imaging system according to one embodiment.

FIGS. 2-21 illustrate a set of user interfaces to configure the thermalimaging system according to one embodiment.

FIGS. 22-24 illustrate a process to identify a location feature in afloor plan used by the thermal imaging system according to oneembodiment.

FIGS. 25-28 illustrate a process to identify a set of parameters forevent detection in the thermal imaging system according to oneembodiment.

FIGS. 29-35 illustrate user interfaces and processes to identify alocation zone in a floor plan used by the thermal imaging systemaccording to one embodiment.

FIG. 36 shows a method to set up a thermal imaging device at a locationfor monitoring according to one embodiment.

FIG. 37 shows a method to establish a scenery model for monitoring alocation according to one embodiment.

FIG. 38 shows a method to validate an event detection model andparameters for monitoring a location according to one embodiment.

FIG. 39 shows a method to configure a thermal imaging system based onuser feedback on notifications of detected events according to oneembodiment.

FIG. 40 shows a data processing system that includes at least a portionof the thermal imaging system according to one embodiment.

DETAILED DESCRIPTION

The following description and drawings are illustrative and are not tobe construed as limiting. Numerous specific details are described toprovide a thorough understanding. However, in certain instances, wellknown or conventional details are not described to avoid obscuring thedescription. References to one or an embodiment in the presentdisclosure are not necessarily references to the same embodiment; and,such references mean at least one.

U.S. patent application Ser. No. 15/607,345, filed May 26, 2017, andU.S. patent application Ser. Nos. 15/797,693 and 15/797,999, both filedOct. 30, 2017, disclose techniques to determine the floor plan or layoutof a location that is being monitored using a thermal imaging device. Anexample of such a location is a room for an elder or a patient. Suchtechniques can be used in the thermal imaging system discussed in thepresent application. The entire disclosures of these applications arehereby incorporated herein by reference.

In general, the floor plan or layout of a location can be used toconfigure the capability of the thermal imaging system in detectingand/or interpreting events from thermal images. Such information can beused as a model of the environment and to understand the environmentaland geographical features and factors of the scenery monitored by athermal imaging device. The environmental model can be used with thermalimages of the monitored location to classify events and controlnotification delivery. The thermal image of a typical room environmentbackground lacks the details for the determination of an environmentalmodel and can look indistinguishably uniform. The floor plan augmentsthe thermal images to facilitate the event detection and classification.User interfaces discussed herein allow the use of user feedbacks and/orthermal inputs generated according to instructions presented in the userinterfaces to determine and/or improve the floor plan and/or other imageprocessing parameters used by the thermal imaging system in detectingand/or interpreting events.

FIG. 1 shows a thermal imaging system according to one embodiment.

The thermal imaging system of FIG. 1 includes a server (115) and athermal imaging device (101) that monitors the environment (107) by theway of thermal imaging. The thermal imaging device (101) is typicallymounted at a fixed location in the environment (107), such as a room.

The Thermal Imaging System (TIS) of FIG. 1 further includes a mobiledevice (105) of a user (103) to provide a graphical user interface toconfigure the thermal imaging system of FIG. 1 , to presentnotifications of detected events in the environment (107), and/or toreceive user feedback on the notifications.

The thermal imaging device (101) and the mobile device (105) can beconnected to the server (115) via a computer communication network(113). The server (115) processes the thermal images captured by thethermal imaging device (101) and provides services based on the thermalimages.

For example, the thermal imaging device (101) can communicate thethermal images to the server (115) via a wireless access point (111) anda computer network (113) (e.g., a local area network and/or theInternet). The mobile device (105), such as a smartphone, a tabletcomputer, a laptop computer, or a personal media player, has a mobileapplication installed therein to communicate with the thermal imagingdevice (101) and/or the server (115) for calibration, setup, and/or theapplication usage of the thermal imaging system.

In some instances, the thermal imaging device (101) communicates thethermal images via a wireless connection, or a wired connection, to themobile device (105), which functions as a host device to furthercommunicate the thermal images to the server (115) for furtherprocessing. The host device, such as the mobile device (105) or anotherdevice in or near the environment (107), can pre-process the thermalimages before providing the processing results to the server (115) forfurther processing and/or for event detection/classification.

The server (115) and/or the thermal imaging device (101) can provide thethermal images to the mobile device (105) for display and/orverification of the event detection/classification.

The server (115) can maintain a user account (121) that facilitatesaccess control and customizes thermal imaging processing in a wayspecific to the environment (107) of the user (103).

For example, the user account (121) can have a device identifier (123)of the thermal imaging device (101) mounted in the environment (107).Further, the user account (121) can have a 2D or 3D floor plan (125) ofthe environment (107). The floor plan (125) identifies location featuresthat specify attributes of regions within the environment. The locationattributes can be used by the server (115) to classify events detectedin the environment (107).

For example, the image processor (129) can use the floor plan (125) todetect events such as person fall on floor, person in bed, person out ofdetection area, person sitting, hazardous hotspot, multiple peoplepresence, person moving, person in a predefined area, smoking detection,fire detection, human close to a hazardous hotspot, water leakage, humaninteraction with other humans, human interaction with other objects,unusual heat patterns of object detection, etc.

For example, the image processor (129) can identify a blob (e.g., a setof pixels) in a thermal image that has a temperature distributiondifferent from the background thermal image and classify the blob as thethermal image of a person or a hotspot. When the thermal image of aperson is detected, the image processor (129) can further classify thepose of the person (e.g., standing, lying down, sitting, walking). Theserver (115) can further interpret/infer the activity of the person fromthe thermal image of the person in view of the location features in thefloor plan (125) of the environment and in view of the pose of theperson and/or the shape of the thermal image of the person. The locationof the person within the environment (107) relative to location featuresof the environment (e.g., path way, activity area, bed, chair, door,window) can be determined based on the line of sight projection of thethermal image relative to the 2D or 3D floor plan (125). The location ofthe person in the thermal image can be used to interpret and/or classifythe event associated with the person.

One aspect of the disclosure discussed herein includes a user interfaceprovided on the mobile device (105). The user interface facilitates thehuman verification and authentication of a particular feature set of thethermal imaging system, where the feature set is configured to detect apredetermined type of events specific to the environment (107). The userinterface is programmed to automatically guide interaction between theuser (103) and the thermal imaging system to test and validate thedetection of a class of events. The human verification andauthentication improves the configuration of the thermal imaging systemand improves the accuracy in event classification and/or notification.

Further, the user interface can provide a feedback or instruction to theuser and guide the user to perform further configuration operations toimprove and/or validate the accuracy of the results generated from thefeature set.

Using the user interface provided in the mobile device (105), a personwithout technical skill can interact with the thermal imaging system toproduce a thermal state in the environment, causing the thermal imagingsystem to analyze the thermal state and generate a notification aboutthe thermal state. The person can then further use the mobile device(105) to validate the authenticity of the state as being identified inthe notification. The user interface guides the user through asimplistic and logical process, whilst the system automaticallyannotates such intervention and stores it in memory to improve its imageprocessing parameters (127) for processing thermal images from thethermal imaging device (101) represented by the device identifier (123)in the user account (121). The improved image processing parameters canbe used by the server (115) to generate improved outcomes in processingsubsequent thermal images generated by the thermal imaging device (101).

The user interface can be used for continued improvements of the imagingparameters (127) using the services of the thermal imaging system. Whenthe server (115) generates a notification of a detected event in theenvironment, the user interface on the mobile device (105) can presentthe notification and receive a user feedback that rates the accuracy ofthe notification. The user feedback can be used to adjust the imagingparameters to improve further event classification and notification inthe user account (121) and decrease false alarms.

For example, after the thermal imaging device (101) captures a thermalimage of the environment (107) using an array of infrared sensingpixels, the server (115) (or the mobile device (105) or another hostdevice, or the thermal imaging device (101)) can use computer visiontechniques to analyze the content of the thermal image. In general, anycomputer vision techniques known in the field can be used to extractcontent of thermal objects from the thermal image. The thermal objectsgenerally have temperature different from the thermal background of theenvironment. Examples of the thermal objects can include persons, pets,stoves, televisions, etc. For example, simple threshold segmentation canbe used to extract objects of certain temperatures above or below athreshold from the thermal background.

Humans typically have a higher temperature than a background. Thus, asegmentation can be applied to extract the thermal images of humans fromthe background. A blob analysis in terms of blob orientation, blobshape, blob neighboring pixels can be performed to extract informationabout a shape, pose, and/or activity of a human. For example, blobproportions may provide insight into the posture of a human.

Some objects can have human-like temperatures, but are typically static.Thus, a temporal analysis of blob movement and blob shape changes(spatial blob changes) can be performed to filter out static “hot spots”such as TVs, PC monitors, home appliances.

For example, an artificial neural network can be trained to classifythermal blobs extracted from the thermal image captured by the thermalimaging device (101).

The thermal imaging system of FIG. 1 can have messaging capabilities.Message triggers (109) can be programmed to send notifications inresponse to a predetermined type of eventsidentified/detected/classified from the thermal images from the thermalimaging device (101). For example, a notification triggered by thedetection of a predetermined type of events, such as person fall, can besent to a user of the account (121) using short message service (SMS),email, push notification, an application programming interface (API),etc. In some instances, the notification can trigger a physical alarm inthe form of sound, siren, light and the like.

For example, in response to the detection of a thermal eventcorresponding to a hazardous hotspot or fire threat alarm, an alarmnotification can be transmitted to an address and/or device identifiedin the user account (121). For example, the event of hazardous hotspotsor fire threat alarms can be programmed to triggered by theidentification of a thermal blob having a temperature above a threshold(e.g., 200 degrees Celsius).

The detection techniques can be implemented in the image processor (129)of the server (115) or in a host device, such as the mobile device(105), or in the thermal imaging device (101).

Using the image processor (129), the server (115) can detect a human(103) in a thermal image captured by thermal imaging device (101) whenthe human enters the environment (107) within the field of view of thethermal imaging device (101). For example, the environment (107) can bea room with services for elders and/or patients.

The server (115) can use the spatial and temporal information about thedetected human (103) to provide human activity data. Such data caninclude human resting, human active, human out of sight, or humanfalling.

For example, to perform human fall detection, the blob of thermal imagethat represents the human (103) can be extracted based on the typicaltemperature range of humans. The server (115) can analyze theorientation and/or shape of the blob to determine whether the human(103) is considered to be standing. For example, when the height towidth aspect ratio of the human blob is higher than a threshold, thehuman (103) can be considered as standing. For example, when thehorizontal extension of the blob is larger by a factor than the blob'sheight, the server (115) can classify the human action/activity aslying, if the human blob is within an area that has a higher than athreshold probability of having a human lying down, according to thefloor plan (125). Examples of such areas include a bed where a personmay rest, an activity area where a person may lie down or fall, a hallway, etc. The server (115) can further use the temporal changes in theshape of the human blob to classify the action/activity of the human(103).

For example, a determination of the human blob representing the human(103) lying in an activity area or hall way can cause the server (115)to determine that an event of human falling has been detected. Theserver (115) can be configured to trigger an alarm and/or a notificationif the human is determined to be in a fall position for a period of timethat is longer than a threshold. For example, a notification can betransmitted to a tablet computer of a nurse to prompt the nurse to takeactions. The notification can identify the environment (107) (e.g., aspecific room) such that the nurse may offer help if needed.

When the thermal imaging device (101) has a low resolution, the thermalmonitoring is non-intrusive and protects the privacy of the occupants ofthe environment (107). However, the low-resolution monitoring can loweraccuracy in activity and/or object detection. When the thermal imaginghas fewer pixels, the image processor (129) has less detail andinformation to work with. For example, a human head can be displayed asa single pixel, making it impossible to detect any facial details, thusmaking it challenging to precisely detect activity or feature patterns.However, at the same time, such low-resolution imaging providesinsufficient detail for human identification and thus improved privacy.Merely a human detection is possible where humans can be detected orrecognized with a certain degree of probability.

For example, the criteria for fall detection can include a horizontalorientation of a human hot-blob. However, the horizontal position of ahuman could be interpreted as a fall, but also, could be a person lyingin bed or on a sofa. A limited resolution has limitations on theaccuracy but is desirable for privacy reasons.

When the thermal imaging device (101) has a low resolution, the mobiledevice (105) can be configured to use user interfaces to obtaininformation to improve the accuracy of the thermal imaging system indetecting events related to humans.

FIGS. 2-21 illustrate examples of user interfaces to configure thethermal imaging system according to one embodiment. The user interfacesare discussed below in connection with the methods of FIGS. 36-40 .

FIG. 36 shows a method to set up a thermal imaging device (101) of athermal imaging system of FIG. 1 in an environment (107).

At block 131, the server (115) creates or registers a user account(121). For example, an installer or a user of the thermal imaging systemcan use the account (121) to interact with the thermal imaging system,such as registering the device identifier (123) of the thermal imagingdevice (101), running a mobile application on the mobile device (105) toaccess services provided via the thermal imaging device, and optionallyusing the mobile application to provide information to generate thefloor plan (125) of the environment (107) monitored by the thermalimaging device (101) and/or provide image processing parameters (127) toconfigure and improve the event detection accuracy in processing thethermal images in the user account (121).

At block 133, the server (115) stores the device identifier (123) tolink a thermal imaging device (101) to the user account (121). Forexample, after the thermal imaging device (101) is physically obtainedand for example unboxed, the user (103) can use the mobile applicationto link the thermal imaging device (101) to the user account (121). Forexample, the mobile device (105) can be a smartphone having a camera andan Internet connection; and the mobile application running in thesmartphone can use its camera to capture or scan a bar code having thedevice identifier (123) of the thermal imaging device (101) to cause theserver (115) to link the thermal imaging device (101) to the useraccount (121). Alternatively, the mobile device (105) can be physicallyconnected via a cable and/or connector to the thermal imaging device(101) (e.g., using a universal serial bus (USB) cable) to link thethermal imaging device (101) to the user account (121); and thecommunication to establish the link between the thermal imaging device(101) and the user account (121) can be made via a wireless and/or wiredcommunication connection between the mobile device (105)/the thermalimaging device (101) and the server (115). Through a user interfaceprovided by the mobile device (105), the thermal imaging device (101) isconfigured to communicate with the server (115) without furtherassistance from the mobile device (105). Thus, after the configurationof the communication connection between the thermal imaging device (101)and the server (115), the mobile device (105) can be physicallydisconnected from the thermal imaging device (101). Furtherconfiguration of the thermal imaging device (101) can be performed via awireless connection or a wired connection. For example, a user interfaceas illustrated in FIG. 2 can prompt the user to capture anidentification code of the thermal imaging device (101) to link thethermal imaging device (101) to the user account (121). Alternatively,the user (103) can use a website of the server (115) to link the thermalimaging device (101) to the user account (121) by entering a serialnumber of the thermal imaging device (101).

At block 135, the server (115) stores data to link the thermal imagingdevice (101) to a location, such as an address of the environment (107)to be monitored by the thermal imaging device (101) and/or the roomnumber or description that identifies the particular location of themonitored area of the thermal imaging device (101) at the address. Forexample, a user interface as illustrated in FIG. 3 can prompt the userto identify the location.

The association between the thermal imaging device (101) and thelocation can be specified and/or modified at any time. However, usingthe user interface, e.g., as illustrated in FIG. 2 to guide the userthrough the process can make the installation procedure very userfriendly and simple.

More than one thermal imaging device can be assigned to monitor the samelocation. For example, a living room may be large and a number ofthermal imaging device (e.g., four) may be needed. For example, eachcorner of a room can have a thermal imaging device installed thereon.

At block 137, the thermal imaging system can instruct the user to powerup the thermal imaging device (101), configure data transmission fromthe thermal imaging device (101) to the server (115), and install thethermal imaging device (101). For example, the data transmission can beconfigured to be performed via a wireless local area network (e.g.,WiFi) that is connected to the Internet, or via a wired connection to aserver (e.g., ethernet). For example, the installation of the thermalimaging device (101) can be performed by simply removing backings ofadhesive tapes pre-installed on the thermal imaging device (101) andattaching the thermal imaging device (101) to one or more surfaces inthe environment. For example, the installation and calibrationtechniques disclosed in U.S. patent application Ser. No. 15/607,345,filed May 26, 2017, can be used. The entire disclosure of the patentapplication is hereby incorporated herein by reference. The installation137 can be repeated for the thermal imaging devices (e.g., 101) used tomonitor the environment (107). Optionally, the operation at block 137can be processed before the operation(s) of block 135 and/or block 133.For example, the powering up of the thermal imaging device can beperformed prior to the linking of the thermal imaging device to the useraccount (133) and/or linking of the thermal imaging device to alocation. In general, the operations of powering up the thermal imagingdevice (101), configuring its data transmission, linking it to a useraccount and linking it to a location can be performed in an arbitraryorder.

Optionally, at block 139, thermal imaging devices (e.g., 101) can begrouped based on scenery and/or rooms. The grouping of devices can bedone prior to system initialization at block 141 so that the user (103)can save time in successive steps.

At block 141, the thermal imaging system performs initialization for thenewly installed thermal imaging devices (e.g., 101). Thus, the server(115) is set up and read for configuration. A user interface asillustrated in FIG. 4 can be used to inform the user (103) of thesuccess in initialization and to guide the user (103) through theconfiguration process.

At block 143, the configuration process can optionally include theselection of features that have predefined event detection services.

At block 145, the configuration process can optionally include thecalibration and verification operations to test and fine tune selectedfeatures of event detection.

In some instances, certain features can be pre-selected; and the user(103) is provided with the user interface to choose from a range offeatures based on the needs of the user and/or adjust the selections.The feature selection (143) can be done post verification (145);however, the verification may be more effective once features are known.

For example, after a user interface as illustrated in FIG. 4acknowledges the successful initialization, a user interface asillustrated in FIG. 5 guides the user (103) to select features from alist. Some features can be included in a certain subscription plan,whilst other features can be add-on services, for which the user (103)can be charged separately. Some features may be free of charge, whilesome features may be offered for a fee and have terms and conditionsattached to them.

The physical installation can include any type of thermal imagingdevices (e.g., 101), without any limitations on resolution, orientationor position of the thermal imaging devices (e.g. 101). Optionally, thethermal imaging devices (e.g., 101) can include other sensors besidesthe image sensors for thermal imaging.

Due to a thermal equilibrium in the room, many room items can have asimilar temperature and emissivity appearing with low or no contrast inthe thermal image of the environment (107). The similar temperaturemakes the items virtually indistinguishable in the thermal band from theroom background. Hence a thermal image of the environment may lookuniform without sufficient information about layout and objects of aroom. Thus, the thermal image cannot be used to determine a floor plan(125) that identifies the location features in the environment (107).Location features provide attributes of sections of areas in theenvironment (107) that can be used to interpret the human activities inthe environment and facilitate event detection.

U.S. patent application Ser. No. 15/607,345, filed May 26, 2017,discloses some techniques to obtain geographic information of thescenery to determine the floor plan (125) and to calculate a referencefor the determination of the dimensions of objects/subjects identifiedin the thermal image.

FIG. 37 shows a method to establish a scenery model for monitoring alocation according to one embodiment.

At block 151, the mobile application running in the mobile device (105)guides the user in capturing reference photographs, as illustrated inFIGS. 6 and 7 .

FIGS. 6 and 7 illustrate user interfaces to use the mobile applicationrunning in the mobile device (105) to guide the user to capturephotographs of the environment (107) for the generation of the floorplan (125). The user is guided by a set of instructions over the userinterface to take one or more reference photographs of the environment(107).

For example, the instructions can ask the user to orientate a cameraassociated with the user (103) and the user account (121) (e.g., thecamera of the mobile device (105), such as a smartphone) in the samedirection as the thermal imaging device (101) and take a reference photofrom such an orientation. For example, if the thermal imaging device(101) is mounted on an edge or corner of a room, the user is instructedto stand in the corner or near the edge to take a picture in anorientation that is substantially consistent with the viewingorientation of the thermal imaging device (101).

For example, the user interfaces provided on the mobile device (105) caninstruct the user to point the camera (e.g., of the mobile device (105))towards a specific region, as illustrated in FIG. 6 . Examples of such aregion is a wall of a room. The server (115) or the mobile applicationrunning in the mobile device (105) can perform a computer visionanalysis to detect boundary lines of the region, such as a wall. Forexample, the live view in the user interface as illustrated in FIG. 7provides a layout indication (301) and a layout annotation (e.g., leftwall) that are overlaid on the live view of the camera for the user tofind such specific region easier. If a particular region is absent fromthe environment, the user interface can instruct the user to capture areference photo of a next region (e.g. right wall, if left wall notseen).

For example, the user interfaces can instruct the user (103) to take apanoramic photograph of the environment from an orientation similar tothat of the thermal imaging device (101). A panoramic photograph can beobtained by panning the camera from one section of the region toanother. Further, the user interface can instruct the user to point thecamera to regions of interests, such as a bed, or a TV or a door.

At block 153, the mobile device (105) transmits the referencephotographs to the server (115).

At block 155, the server (115) and/or the mobile application running inthe mobile device (105) can analyze the features, content, regions andgeography in the reference photographs.

At block 157, the server (115) and/or the mobile application can extractthe features, content, regions and geography from the referencephotographs to establish a scenery model, such as the floor plan (125).

At block 159, the server (115) stores the scenery model in associationwith the thermal imaging module (101).

For example, the reference photographs can be analyzed to identifylocation features (e.g., regions of predefined types), such as beds,windows, doors, lights, appliances, furniture, etc. Such features aretypically visible and identifiable in the photographs captured in lightbands visible to human eyes and by the camera of the mobile device(105). In some lighting conditions, location features with bright andhigh intensity can be identified easier than other features. Thelocation features can be extrapolated into a 3D space model to generatethe floor plan (125) that annotates different sections of theenvironment with attributes that can be used for event detection,identification, and/or classification.

In some instances, the user interface provided by the mobile applicationcan include an augmented reality (AR) ruler and measurement tool. Thus,some geometric features can be extracted from the visual photograph,either automatically or with the aid of inputs provided by the user(103).

Alternatively, the user can have an option to configure a blank orpre-set 3D model space with features, and build a 3D model of theenvironment (107) using the mobile application.

The visual references generated using the user interfaces of the mobileapplication can be limited in accuracy, due to the unknown distance andorientation between the location of photography by user and the actualmodule placement. Further, the computer vision and augmented reality(AR) tool may have limitations to its detection accuracy. However, thevisual references can provide a valuable first approximation of thelocation features in the floor plan (125) of the environment (107). Theprocess can be repeated to increase the accuracy of the model builtusing the visual references. Thus, the thermal imaging device (101) doesnot require a visual camera for capturing images in lights visible tohuman eyes. Including a camera in the thermal imaging device (101) canraise privacy concerns. The visual referencing is performed using analien tool (e.g., the mobile device (105) that is not part of thethermal imaging device (101) during the normal operation/monitoring ofthe environment (107)). The performance of the visual referencing islimited to the installation/configuration process. Hence privacyconcerns can be reduced or eliminated. The visual referencing does notrequire lots of effort from the user (103). It is designed to be userfriendly and take only a few moments, and is cost effective as thecamera of the mobile device (105) is used, which is already available.No additional camera is required to perform the task of visualreferencing. Should layout of the room change over time (e.g., movingbed or furniture, or the thermal imaging device (101)), then the visualreferencing can be repeated. In such an instance of layout changes, theuser interface provided on the mobile device (105) can prompt the userto re-capture visual references, such as when the image processing inthe system does not correlate with the references that have beenconfigured previously.

For example, the server (115) and/or the image processor (129) can beconfigured to: guide the user (103) to capture reference photographs(e.g., as illustrated in FIGS. 6 and 7 ), analyze the photographs toidentify location features and construct the floor plan (125) (with orwithout persistent storage of the photographs) using visual imageprocessing techniques and tools such as AR measurement tool, extractlocation features, content, regions and geography and generate a 3Dmodel of the environment, and store the model as the floor plan (125)with location information associated with the identifier (123) of thethermal imaging device (101).

The server (115) can use the floor plan (125) to classify events. Forexample, in response to a determination that a human (103) is in ahorizontal position, the server (115) can classify the event as “humanin bed” when the human (103) as detected in a thermal image is locatedin a bed region, and classify the event as “human falling” when thedetected human (103) is located in a hall way or an activity area.

In some instances, the floor plans (e.g., 125) of multiple rooms canlook similar. For example, an elder care facility may have a same floorplan for many rooms. Hence the floor plan (125) of one room can be usedfor other rooms, by simply referencing or copying the floor plan (125).In some instances, a library of pre-built floor plans can be presentedfor select as a starting point for building the floor plan (125) for theenvironment (107). The floor plan setting can be optional in theinitialization process of the Thermal Imaging System (TIS).

FIG. 38 shows a method to validate an event detection model andparameters for monitoring a location according to one embodiment.

After establishing referencing and calibrating parameters, the server(115) is operational in classifying events based on the thermal imagesfrom the thermal imaging device (101). The method of 31 can be used totest and/or validate the detection features of the thermal imagingsystem. The verification can be done optionally or as part of thecalibration and configuration process.

The general problem of computer vision techniques is that 100% accuracycannot be provided or guaranteed. Some small unknown factors such aspartial occlusion, changes in the scenery or unexpected features candisrupt computer vision and provide a false outcome.

To overcome such shortcomings, the mobile application running in themobile device (105) can provide a user interface to facilitate a humanverification/validation of certain events detected by the thermalimaging system. The validation can be used to verify whether the systemcan detect an event that should be detected by the system, whether anevent reported to be detected by the system is actually present in theenvironment (107) (e.g., false positive), and/or whether an eventdetected by the system is correctly classified. The user interfaceallows the user (103) to provide feedback to the thermal imaging systemto annotate its detecting results and thus improve the accuracy of thesystem. Such a method to improve the accuracy of the thermal imagingsystem can be very valuable. Over time and over a number of feedbackloops the system can refine and offer highly accurate results.

At block 161, the method of FIG. 38 initiates a verification process.

At block 163, the mobile application instructs the user to enter aposition at a location. Such location can be identified using the floorplan (125). In general, it is optional to perform the operations offloor plan determination. For example, to perform “in bed detection” thepanoramic photo may or may not be obtained for floor plan determinationprior to the services of “in bed detection”.

At block 165, the thermal imaging system processes a thermal image todetect an event consistent with the user being in the position at thelocation (165).

At block 167, the mobile application determines whether the thermalimaging system has detected the event.

At block 169, the mobile application provides a confirmation of thedetection based on a user input to the mobile application, when thethermal imaging system is able to detect the event.

At block 169, the mobile application provides instructions to the userto further configure the system, when the thermal imaging system is notable to detect the event consistent with the user being in the positionat the location (165).

For example, FIG. 8 illustrates a user interface to start a verificationprocess. Each detection feature can be verified, one-by-one, or it canbe skipped or verified at a later point in time. For example, for thefeature in bed detection, a subject is instructed to lay in the bed, asillustrated in FIG. 9 . The subject can be either the user (103) orsomeone else.

If the thermal imaging system is able to detect the event of human inbed, the user interface illustrated in FIG. 9 reports the detectionusing the message “In bed detected” and allows the user to provide aconfirmation by selecting the “Verify” button in FIG. 10 . The message“In bed detected” can be highlighted (e.g., via color scheme, font,and/or animation). For example, the detection can be performed based ondetecting a blob in the of thermal image having a temperature in therange of the body temperature of humans, and based on a determinationthat the blob is in a horizontal orientation, and optionally the blob isin a bed area identified by the floor plan (125).

Optionally or in combination, an acoustic signal or notification can beplayed to indicate the successful detection. If the detection isaccurate and true, the user interface prompts the user to provide afeedback, such as clicking the “Verify” button illustrated in FIG. 10 .

Once the human verification is received in the thermal imaging system,the server (115) can stored the position and/or other characteristics ofthe human blob extracted from the thermal image as the image processingparameters (127) associated with the verified detection feature (e.g.,human in bed). Thus, when a subsequent detection matches the imageprocessing parameters, the likelihood of an accurate detection isimproved.

FIG. 22 illustrates an example of a low resolution thermal image havinga blob (303) of a human subject in an annotated bed area (305). Thesubject can be highlighted by default background subtraction. Thebackground appears black in the image of FIG. 22 , whilst the subjectrepresented by the blob (303) has a grayscale and contrast against thebackground. The subject's position can be determined based on its blobsize, shape and orientation. In this example, the orientation of thesubject is horizontal; and the thermal imaging system determines thatthe subject is in bed and lying. From the orientation and size of theblob (303) representing the human subject, the server (115) can estimatethe perimeter of the bed area (305), which can be used to augment thefloor plan (125) and/or can be stored as part of the image processingparameters (127).

In some instances, the detection result message (e.g., “In beddetected”) may not be accurate or true. For example, a subject comesclose to the bed, or sits down on a sofa close to the bed; and inresponse, the thermal imaging system may mistakenly conclude that thesubject is in bed and thus provide the false indication of the detectionresult. In such a situation, the user interface illustrated in FIG. 10allows the user to select “Bed unoccupied” to provide a feedback tocorrect the thermal imaging system. The feedback can overwrite aparameter associated with the detection of human in bed. For example,the server (115) can mark in the floor plan (125) the region occupied bythe human blob as “not bed”, such that when a human blob is detected inthe area again, the server (115) does not classify the presence of thehuman in the area as human in bed.

In other instances, when a subject is in bed, as instructed via the userinterface of FIG. 11 , but the thermal imaging system fails to determinethat the thermal image is showing a human in bed, the user interface ofFIG. 12 can be presented after a period of time to indicate that thesystem failed to detect the event. When the user selects the button “Iam in bed but no alarm indication showed up”, a user interface asillustrated in FIG. 13 can be presented to prompt the user to specifywhether a human subject is currently in bed or no human subject iscurrently in bed. A selection made by the user in the user interface ofFIG. 13 can be used by the server (115) to adjust the floor plan (125)and/or the image processing parameters (127) to improve the detectioncalculation.

For example, if the user confirms that a subject is in bed by clickingthe button “subject in bed”, the parameters of human blob shape, humanblob type, human blob orientation and location can be used to adjustcomputer vision computation to arrive at the conclusion that the blobrepresenting a human in a horizontal position, and/or the floor plan canbe adjusted to show that the location of the blob is a bed area. Thus,this specific shape, location, orientation, size and temperature profileof the blob shall be associated for positive identification of human inbed.

Due to the orientation and possibly the imaging distortion of thethermal imaging device (101), a blob showing a subject lying can haveproportions similar to a blob showing a subject standing. Theverification process as illustrated above using the user interfaces ofFIGS. 11-13 allows the thermal imaging system to fine tune the detectionparameters to generate accurate results.

For example, when the user selects the button “subject in bed” in FIG.13 , the server (115) can adjust or customize the size, shape, ratiothresholds for orientation classification for the region occupied by theblob (303) and for the thermal image device (101) to allow the system toreach a conclusion that the human subject is lying and thus facilitatethe determination of human in bed.

FIGS. 22-24 show a human subject on a bed in different positions. Themobile application can instruct the user to take different positions inthe bed to provide the thermal imaging system with parameters that canbe used to improve the capability of the thermal imaging system indetect human in bed with different positions.

A set of user interfaces can guide the user to refine the detectioncapability of the thermal imaging system by using verification eventsthat cover a variety of scenarios, as illustrated in FIGS. 14-16 .

The user interfaces in FIGS. 14-16 instruct the user (103) to changepositions once or multiple times for the validation/refinement of humanin bed detection. The user can change positions in the bed in a way asillustrated in FIGS. 22-24 . From the images illustrated in FIGS. 22-24that are identified by the user (103) as showing human in bed, theserver (115) collects different sets of estimations of the bed perimeterand the blob characteristics representing human in bed. Thus, the servercan not only mark the associated positions as “in bed”/lying, but alsocan refine the location of the bed perimeter more precisely fromcombining the bed perimeters estimated from the different lyingpositions of the subject in the bed. The subject is not required to havetechnical skills or technical knowledge to train the thermal imagingsystem. By simply asking the subject to change position in the bed, thethermal imaging system can obtain a set of inputs to refine theparameters for event detection and/or classification. Thus, the userinterface is very user friendly and effective in improving theperformance of the thermal imaging system.

Upon human verification the user interface can instruct the subject toexit the bed and to re-verify placing a subject, him or herself backinto bed, as illustrated in FIG. 17 . With the modified/customized floorplan (125) and the improved image processing parameters (127), thethermal imaging system can detect the event correctly and show thedetection result message “In bed detected”, as illustrated in FIG. 9 ,during the re-verification process.

The refinement process illustrated in FIGS. 14-16 is particular helpfulif there is a large offset in the detection results made using defaultsettings (e.g., human identified as standing whilst lying in bed) or ifa second verification run is unsuccessful (e.g., no in bed detected eventhough the user explicitly confirms that a human subject is actually inbe). The refinement process allows the thermal imaging system to gainhigher accuracy.

FIGS. 18-19 illustrate a set of user interface to instruct the user(103) to perform a sequence of activities to generate thermal inputs torefine the event detection capability of the thermal imaging system.

The user interface of FIG. 18 instructs the user to walk around the bed.As a result of the sequence of activities of walking around the bed, thethermal image system captures a set of thermal images as illustrated inFIGS. 25-28 . Since the thermal images show a human walking around abed, the server (115) can infer the perimeter of the bed from the movingblob of the user as captured in the thermal images.

In general, the system can instruct the user to position himself orherself in the vicinity of certain location features to annotate thelocation features, such as a bed, or a sofa or a table, or furniture,etc.

In case of the bed, the user can be instructed to walk around the bed toallow the thermal imaging system to estimate the perimeter of the bed.Further, the blobs of the user walking around the bed represent the userin a standing/vertical position. Comparing the characteristics of blobsof the user in a standing/vertical position and characteristics of blobsof users in a lying/horizontal position allows the server (115) toderive/refine parameters to classifying a blob orientation (e.g.,standing/vertical, or lying/horizontal).

For example, from the thermal images of the user walking around the bed,the server (115) can mark and store the locations of the blob andidentify occlusions of the blobs. The occlusions can be associated withthe portions of the bed between the line of sight of the occludedportions of the user and the thermal imaging device (101). Further foreach frame obtained of the user walking close or around the bed, theblob parameters can be stored as the image processing parameters (127);and the parameters can be references in determination whether a human isstanding close to or next to the bed, or lying in the bed.

For example, the user interface of FIG. 18 can show a message “detecting. . . ” until the user (103) finishes walking around the bed.Automatically, the server (115) draws a perimeter/geographical locationmarker next to the blob of the human in the thermal image, following thehuman. At the first turn of the human the server (115) can determinewhether the initial line shall be to the left or right of human. As thehuman walks the perimeter of the bed, the server (115) marks thelocation. Once the user (103) has finished walking around the perimeteror walking partially around perimeter (e.g. if bed against wall, onlyone, two or three sides of bed can be walked along), the server (115)can detect the completion of the walk and provide the acknowledgmentmessage illustrated in the user interface of FIG. 19 . Optionally, themobile application can show the thermal images as illustrated in FIGS.25-28 while the user (103) walks around the bed.

In some instances, if the bed has only one side to walk along, the usercan stop the process, by selecting the link “Finished walking”. Partialperimeters can be automatically closed by the server (115) through aclosing or estimation procedure (e.g., close a polyline).

In general, a verification flow of a detection feature of the thermalprocessing image can include: initiating a verification process,checking whether the verification is successful or not; if theverification is successful, annotating parameters (e.g. defaultparameters as correct) and provide a verification/confirmation messageto the user; and if the verification is not successful, prompting humaninteraction, where a human user (103) can manually annotate theevent/feature, or provide thermal inputs to refine the accuracy of thefeature. The verification flow can be repeated until the verification issuccessful.

For example, the user can be instructed to walk out of the room for “outof room detection”. In such an event, a human blob disappears out of thescenery captured by the thermal image of the imaging device (101). If ahuman is detected inside the room, whilst in reality the human isoutside of the room, which can be possible as a result of the defaultimage processing parameters causing the server (115) to mistakenlyidentify a static hot spot as a blob of a human. For example, the humanheat residue (“heat shadow”) from sitting on a couch or bed may lead tothe thermal imaging system to incorrectly conclude that a human is onthe couch or bed. In such a situation, the user interface provided onthe mobile device (103) can instruct the user (103) to confirm that heor she is outside of the room (e.g., environment (107)) and no one is inthe room. The user confirmation allows the server (115) to mark thehot-blobs in the room as “non-human” or “static”, to betterdifferentiate between human and static hot blobs. Further, in case of a“heat shadow”, the server (115) can stored the location of the heat blob“separating” from a human blob and mark these locations of zones wherebody heat can be transferred to objects and to denote these heatsignatures as non-human, even though they may have human shape, size,orientation and heat signature for a limited amount of time. Further, ifthere shall still be an undefined state detected by the server (115),such as a “non-human hotspot” being identified as human by the server(115) while the user confirms no one in the room, the server (115) canfurther instruct user, as in the second refinement procedure describedabove, for the user to walk to the entrance/exit door, and possibly walkinto the room/scenery, until the user is detected by thermal imagingdevice (101) and then the server (115) can mark a certain location andscenario as entrance/exit.

For example, the user can be instructed to lay on the floor to verify a“fall detection” feature. Such a feature can be very useful for agedcare. The user interface on the mobile device (105) can indicatedetection status on a person lying on the floor, and if no detection isindicated then the user would be instructed to verify throughinteraction. Fall detection may be more complex than in bed detection orout of room detection, as a) there are a vast amount of positions andlocations where a fall can occur within the field of view of the thermalimaging device (101), b) a fall could be partially or fully occluded byobjects in the room making the fall hard to detect, and c) the fallcould occur in areas where there is no line of sight and field of viewof the thermal imaging device (101), hence undetectable. Therefore, falldetection may be harder to achieve the same level of accuracy for otherdetection features offered by the thermal imaging system. On the otherhand, instructing the user to simulate all possible fall scenarios,positions and locations would be not user friendly. Therefore, thethermal imaging system may use its default settings for thecommencements of its operation, which can provide false positives andindicate falls, even if the outcome is not identified (meta-state,unknown state between e.g. fall and standing). False positives arealarms or notifications of fall detection, where a fall is identified bythe thermal imaging system but, in reality, may not have been a fall.

For example, the user can be instructed to go to a particular region orzone of interest using a user interface illustrated in FIG. 29 . A userinterface illustrated in FIG. 30 contains an “Approve” button, which canbe activated by the user once the user is in the particular region orzone of interest. When the “Approve” button is activated, the thermalimaging device (101) identifies the blob of the image (307) of the userwithin the thermal image (313) of the environment (107) (e.g., asillustrated in FIG. 33 and captured by the thermal imaging device (101))and uses the image (307) of the user to generate an estimated size andlocation of the zone. For example, a user interface illustrated in FIG.31 starts a process of identifying the size, location and/or shape ofthe zone (e.g., a bed, an activity area, a hall way, a dining area). Theboundary (309) of the image (307) of the user as illustrated in FIG. 34can be used as an initial estimate of the boundary of the zone. Theestimation can be presented in a user interface as illustrated in FIG.35 to allow the user to adjust the estimation. For example, the user mayuse a finger (311) to touch and select the estimated boundary on a touchscreening showing the thermal image (313) of the environment (107),causing the thermal imaging system to adjust the location and size ofthe estimated boundary. For example, the estimated boundary (309) can bedragged over the image (313) to adjust its location. For example, theestimated boundary (309) can be scaled to increase or reduce its size.In some instances, the finger (311) may draw a shape (309) around theimage (307) of the user presented on a touch screen to specify theboundary of the zone. Alternatively, or in combination, the user may usehis/her image (307) as a paint brush tool to paint the zone in thethermal image of the environment (107), by moving around in the zonewhile the user interface of FIG. 31 is displaced; and the area in theimage (313) painted using the paint brush tool identifies the shape andlocation of the zone. A user interface of FIG. 33 has an “Approve”button that can be activated to confirm the user acceptance of thelocation, shape and/or size of the boundary (309) of the zone aspresented with the thermal image (313) of the environment (107).

User annotations to identify location features can be generated not onlybased on the thermal image (307) of the user extracted from the thermalimage (313) of the environment (107) captured by the thermal imagingdevice (101), but also the thermal images of other objects. The thermalimaging system of FIG. 1 can identify and/or extract hot and/or coldobjects that have temperatures different from the background temperatureof the environment (107). Examples of such objects include modems,computers, television sets (TVs), refrigerators, stoves, and appliances.The thermal imaging device (101) can identify the blobs of thermalimages of the objects within the thermal image (313) of the environment(107) and requests the user to annotate or classify the objects. Forexample, a user interface may highlight a thermal image blob extractedfrom the thermal image of the environment (107) and present a pull-downmenu that allows the user to select a characterization/identification ofthe object. When the thermal image blob matches a predetermined,distinct heat signature of a type of objects (e.g., modems), the thermalimaging system of FIG. 1 can automatically annotate the object with orwithout user confirmation. Some objects are cold or hot spots, such aswindows, doors, sinks, and objects of different emissivity ortemperature from the overall background temperature of the environment(107).

Sun irradiation can create a hot spot in the environment (107). This hotspot could be interpreted as a human as it may move very slowly andcould have human like shape in a low resolution thermal image.Optionally, this hot spot created by sun irradiation can be annotated bythe user. Alternatively or in combination, the thermal imaging system ofFIG. 1 can use weather data of the geographical area of the environment(107) to determine whether the host spot is a result of sun irradiation.For example, the travel path of the sun hot spot relative to theorientation of the room and the location of the thermal imaging device(101) can be calculated to determine whether there is a match betweenthe calculated path the observed location/path of the hot spot. Thedegree of matching can be used to determine whether the hot spot is aresult of sun irradiation. A compass sensor in the mobile device (105)during setup can be used to identify the relative orientation of thescenery relative to the direction of north and thus facilitate theidentification of hot spots generated by sun irradiation.

Weather data can be used to determine whether an air conditioner (AC) ora heater is likely being operated in the environment (107) and thus seenin the thermal image (313) of the environment (107). For example, if aheater is detected in thermal imaging system of FIG. 1 while the outsideair temperature is in a range that typically causes the use of heaters,it can be determined that the presence of the operating heater isnothing out of the ordinary. On the contrary, if outside air temperatureis hot (e.g., 100-degree Fahrenheit) and the heater is operating insidethe room to increase the background temperature in the room, the thermalimaging system of FIG. 1 can send a notification or alert to the user(103) and/or other recipients specified in the user account (121).Weather data can be useful in determining whether the backgroundtemperature of the environment (107) is within a standard range suitablefor normal living. An alert or notification can be generated when thebackground temperature of the environment (107) is outside of the range.

The thermal imaging device (101) can optionally include other sensors,such as time of flight sensors, microphone, lidar, etc. For example, themicrophone and a speaker in the thermal imaging device (101) can be usedto facilitate communications between a human administrator (or acomputerized administrator) and a person in the environment (107) inresolving false alarms and/or generating user annotations. For example,when the environment (107) is monitored for elder/patient care, thethermal images captured by the thermal imaging device (101) can beanalyzed to detect a possible abnormal situation/emergency, such as afall. A nurse can be notified of the situation to initiate acommunication and/or obtain a feedback from the patient/resident. Usinga communication link to the thermal imaging device (101), the nurse canoperate a mobile applicating running in a mobile device to check andengage with person using voice, asking them if they are ok? If theperson responds: “I am ok”, the situation can be annotated and/or usedto adjust subsequent responses for a similar subsequent detection. Insome instances, a computerized the attendant can initiate a similarvoice communication through text to speech synthesizer. The voiceresponse from the person (patient/resident) can be analyzedautomatically using a voice recognition technique to determine aresponse. In some instances, an artificial intelligence system can beimplemented in the server (115) to process the voice response from theperson in the environment and/or formulate a response, such as cancelinga false alarm, annotating a false alarm, calling a nurse and/or anotherregistered person for assistance, etc. Valuable time can be saved whenthe nurse doesn't need to run down to the room every time there is afall detection alarm. In some instances, the voice interface can beimplemented using a separate device connected to the server (115). Forexample, the separate device can be a mobile device (105), a smartspeaker, a smart television set, a smart home hub/controller, etc. thatreceives voice input from the person in the environment (107) andprovides voice response to the person.

The mobile device (105) can provide a user interface that allows theuser (103) to annotate a fall detection notification. For example, ifthe server (115) identifies a fall and generates a notification and theuser (103) determines that there is actually no fall, the notificationis a false positive; and the user interface of the mobile device (105)allows the user (103) to provide a feedback rating the notification as afalse positive, which causes the server (115) to annotate the event(and/or the parameters of the thermal blob that causes the falsenotification).

FIG. 39 shows a method to configure a thermal imaging system of FIG. 1based on user feedback on notifications of detected events according toone embodiment.

At block 181, the thermal imaging system detects an event fromprocessing a thermal image captured by a thermal imaging device (101)mounted to monitor the environment (107).

At block 183, the server (115) of the thermal imaging systemcommunicates the event to the user (103) through a notification to amobile device (105). A user interface illustrated in FIG. 20 presentsthe notification and allows the user (103) to provide feedback about theaccuracy of the notification and/or the detection of the event.

For example, when the server (115) extracts a human blob having a shape,location, size that cannot be classified as a known non-fall position,such as standing, the server (115) can classify the event as “humanfalling” and generate a “fall detected” notification message to themobile device (105) registered in the user account (121).

Alternatively, or in combination, the notification can be sent toanother device (e.g., siren, speaker, etc.) or to another configureduser.

A mobile application in the mobile device (103) can provide the userinterface as illustrated in FIG. 20 to receive user input on theaccuracy feedback on the notification. Alternatively, or in combination,a website of the server (115) can be used to provide a user interface toreceive the user feedback.

At block 185, the server (115) receives an input from the user (103)regarding confirmation of the event reported by the notification.

At block 187, the server (115) determines whether the input confirms theoccurrence of the event, or identifies the notification as falsepositive.

At block 189, if the user input confirms the occurrence of the event asreported by the notification, the server (115) stores data annotatingthe validity of the detection; and subsequently, at block 191, theserver (115) can transmit a notification to the user (103) when theevent is re-detected from a subsequent thermal image.

At block 193, if the user input identifies the notification as falsepositive, the server (115) stores data annotating the invalidity of thedetection; and subsequently, at block 195, the server (115) can suppressa notification to the user (103) when the event is re-detected from asubsequent thermal image.

For example, the user interface of FIG. 20 has a button “Mute” that canbe selected to acknowledge the notification, and a button “This was nota real fall” that can be selected to indicate to the server (115) thatthe alarm/notification is “false positive”.

For example, in response to the user selecting the button “This was nota real fall”, the shape, features, location and/or other parametersassociated with the human blob that triggers the notification can beidentified as not associated with fall, such that a future detection ofa human blob having a same or similar shape, features, location and/orother parameters can be classified as “not-falling”.

For example, in response to the user selecting the button “Mute” withoutselecting the button “This was not a real fall”, the server (115) canannotate the shape, features, location and/or other parametersassociated with the human blob that triggers the notification as beingassociated with “real fall”.

The annotated parameters can improve the accuracy of the server (115) inclassifying the events detected in the thermal image from the imagingdevice (101).

The server (115) can improve its accuracy in event classification byprogressing the parameters associated with the annotated eventnotifications.

For example, if a blob of a certain shape is marked as a “real fall”,slight aberrations/variation/deviations of its parameters can also bemarked for future reference as falls as well.

In some instances, the server (115) can detect a human blob in a fallposition and subsequently determines that the human blob starts moving(e.g., walking out of the door). Such a situation generally does notcorrespond to a fall of an elder or patient that results in anemergency. Thus, the server (115) can adjust its notification policysuch that if such a human blob posture occurs in future, the server(115) can delay the transmission of the notification/alarm and wait fornext action of the human blob to determine whether the fall is anemergency.

Fall detection can be also improved by taking certain locations andregions of the scenery into account. For example, a fall next to a bedcan be more likely than in the middle of the room. Therefore, thedetection of the bed can be helpful for identifying falls in vicinity ofthe bed.

In some implementations, when a human blob becomes partly occluded by anobject, the server (115) cannot, by default, assign a definite state tothe human represented by the blob. In such a situation, the server (115)can identify the human blob as in a meta-state or unknown state. Toimplement a conservative notification policy, the server (115) can senda fall indication, which allows a user to annotate the detection usingthe user interface illustrated in FIG. 20 . The user feedback assiststhe server (115) in classifying the state of the human represented bythe blob and to improve the floor plan 127 (e.g. occlusion refinement ornew objects which create occlusion).

In some implementations, when a human blob can be classified to be inmultiple states. To implement a conservative notification policy, theserver (115) can send for example a fall indication, which allows a userto annotate the detection using the user interface illustrated in FIG.20 . The user feedback assists the server (115) in classifying the stateof the human represented by the blob. Further, the thermal imagingsystem can send notifications to users to help improve unknown states.

In some implementations, when a false positive is identified where asubject is partially occluded, the user interface provides an option toadd information about the occlusion. For example, the user interfaceillustrated in FIG. 21 can provide the message: “Not a real fall (falsepositive) Thank you for your feedback. Has the location layout changed(e.g., bed moved to new location)?”. The button “Yes” and the button“No” in FIG. 21 can be used by the user to indicate whether the layouthas been changed.

In some instances, the message from the server (115) can state that:“The alert is given because the subject was (partially) occluded. Couldyou provide information about the object of occlusion?”. The userinterface is then configured to allow the user (103) to make a binarychoice for either “Yes” or “No”.

If the user indicates a location layout has been made in theenvironment, a dropdown list, for example, can be presented with typicalitems such as sofa, bed, cabinet, chair, table, etc. to identify theitem that is involved in the layout change; and the user (103) has anoption to manually name an item for the layout, if the item is not foundin the default list. The user interface can receive information aboutlocation and size (e.g., 2-meter-tall cabinet left of bed, approx.1-meter wide). Such information can be stored in the floor plan (125)and used in the event detection computation/classification (e.g.,occlusion processing). Further, the user may input the information tothe mobile device (105) via voice command and the mobile applicationrunning in the mobile device (105) can use speech recognition to obtainand extract relevant information. The user can for example say “bed hasbeen moved 2 meters from previous position and a new drawer has beenadded left of bed”; and in response, the mobile application extract theinformation and annotate it in the parameters (127).

In general, the discussed above feedback mechanism can be used with anydetection/notification feature of the thermal imaging system. Forexample, the thermal imaging system can implement a hazardous hotspotdetection feature. When a hotspot is detected by the thermal imagingsystem, the server (115) can generate a “hazardous hotspot detected”notification/alarm. The user interface implemented in the mobile device(105) can receive user feedback as to whether the detected hot spot is“good” or “bad”. A “good” hotspot object can occur in the day to daylife in the environment, such as a stove, hot plate, microwave, etc. Thetemperature of the “hotspot” object can exceed a safety temperaturerange for a human (e.g., a temperature above 50 degrees Celsius cancause burns). A “bad” hotspot can be an actual threat and action fromthe user would be required to eliminate it.

If the user (103) classifies a hotspot as “good”, a re-occurrence of thehotspot with its exact same or similar parameters (shape, size,location, temperature profile) can be classified as non-hazardous; andthus, the server (115) can suppress a “hazardous hotspot detected”notification/alarm in response to the re-occurrence.

If the user (103) classifies the hotspot as “bad”, the re-occurrence ofthe hotspot and/or similar occurrences can be classified as hazardousand thus can trigger the “hazardous hotspot detected”notification/alarm.

In one scenario, a bed in the environment (107) has been moved, whichcan occur in facilities hosting patients and/or elders. A resident lyingin the bed at a different position, which has been previously verified,could trigger a fall detection alarm. The user (103) (e.g., nurse) couldthen provide feedback (e.g., using the user interfaces of FIGS. 20 and21 ) to identify the false positive and provide information to adjustthe floor plan (125) and thus prevent further false positivenotifications.

The human annotations of features and events can be stored in the server(115). The server (115) can apply sophisticated computation techniquesto improve its detection and/or classification capabilities. Forexample, the human annotations identify the desirable classificationresults; and thus, a supervised machine learning technique can beapplied to train an Artificial Neural Network (ANN) to perform eventclassifications. For example, the annotations from different useraccounts can be aggregated to train a general ANN model for eventclassification for a set of user accounts. The general ANN model can beused as a default model for a user account (121); and the annotations inthe particular user account (121) can be used to further train the ANNmodel to generate a customized ANN model for the environment (107).Further new firmware updates on the server can be implemented with moretrained, more accurate ANN models.

The present disclosure includes the methods discussed above, computingapparatuses configured to perform methods, and computer storage mediastoring instructions which when executed on the computing apparatusescauses the computing apparatuses to perform the methods.

In FIG. 1 , each of the mobile device (105), the server (115), and thethermal imaging device (101) can be implemented at least in part in theform of one or more data processing systems, with more or fewercomponents.

FIG. 40 shows a data processing system that includes at least a portionof the thermal imaging system according to one embodiment.

FIG. 40 shows a data processing system that can be used to implementsome components of embodiments of the present application. While FIG. 40illustrates various components of a computer system, it is not intendedto represent any particular architecture or manner of interconnectingthe components. Other systems that have fewer or more components thanthose shown in FIG. 40 can also be used.

In FIG. 40 , the data processing system (200) includes an inter-connect(201) (e.g., bus and system core logic), which interconnects amicroprocessor(s) (203) and memory (211). The microprocessor (203) iscoupled to cache memory (209) in the example of FIG. 40 .

In FIG. 40 , the inter-connect (201) interconnects the microprocessor(s)(203) and the memory (211) together and also interconnects them toinput/output (I/O) device(s) (205) via I/O controller(s) (207). I/Odevices (205) may include a display device and/or peripheral devices,such as mice, keyboards, modems, network interfaces, printers, scanners,video cameras and other devices known in the art. When the dataprocessing system is a server system, some of the I/O devices (205),such as printers, scanners, mice, and/or keyboards, are optional.

In FIG. 40 , the memory (211) stores a thermal imaging system (TIS)application (213). For example, the TIS application (213) can be amobile application implemented in the mobile device (105). For example,the TIS application (213) can be a set of instructions implementing theimage processor (129) of the server (115). In some instances, thefunctions of the TIS application is implemented at least in part vialogic circuits, such as Application-Specific Integrated Circuit (ASIC)or Field-Programmable Gate Array (FPGA).

The inter-connect (201) includes one or more buses connected to oneanother through various bridges, controllers and/or adapters. Forexample, the I/O controllers (207) include a USB (Universal Serial Bus)adapter for controlling USB peripherals, and/or an IEEE-1394 bus adapterfor controlling IEEE-1394 peripherals.

The memory (211) includes one or more of: ROM (Read Only Memory),volatile RAM (Random Access Memory), and non-volatile memory, such ashard drive, flash memory, etc.

Volatile RAM is typically implemented as dynamic RAM (DRAM) whichrequires power continually in order to refresh or maintain the data inthe memory. Non-volatile memory is typically a magnetic hard drive, amagnetic optical drive, an optical drive (e.g., a DVD RAM), or othertype of memory system which maintains data even after power is removedfrom the system. The non-volatile memory may also be a random accessmemory.

The non-volatile memory can be a local device coupled directly to therest of the components in the data processing system. A non-volatilememory that is remote from the system, such as a network storage devicecoupled to the data processing system through a network interface suchas a modem or Ethernet interface, can also be used.

In this description, some functions and operations are described asbeing performed by or caused by software code to simplify description.However, such expressions are also used to specify that the functionsresult from execution of the code/instructions by a processor, such as amicroprocessor.

Alternatively, or in combination, the functions and operations asdescribed here can be implemented using special purpose circuitry, withor without software instructions, such as using Application-SpecificIntegrated Circuit (ASIC) or Field-Programmable Gate Array (FPGA).Embodiments can be implemented using hardwired circuitry withoutsoftware instructions, or in combination with software instructions.Thus, the techniques are limited neither to any specific combination ofhardware circuitry and software, nor to any particular source for theinstructions executed by the data processing system.

While one embodiment can be implemented in fully functioning computersand computer systems, various embodiments are capable of beingdistributed as a computing product in a variety of forms and are capableof being applied regardless of the particular type of machine orcomputer-readable media used to actually effect the distribution.

At least some aspects disclosed can be embodied, at least in part, insoftware. That is, the techniques may be carried out in a computersystem or other data processing system in response to its processor,such as a microprocessor, executing sequences of instructions containedin a memory, such as ROM, volatile RAM, non-volatile memory, cache or aremote storage device.

Routines executed to implement the embodiments may be implemented aspart of an operating system or a specific application, component,program, object, module or sequence of instructions referred to as“computer programs.” The computer programs typically include one or moreinstructions set at various times in various memory and storage devicesin a computer, and that, when read and executed by one or moreprocessors in a computer, cause the computer to perform operationsnecessary to execute elements involving the various aspects.

A machine readable medium can be used to store software and data whichwhen executed by a data processing system causes the system to performvarious methods. The executable software and data may be stored invarious places including for example ROM, volatile RAM, non-volatilememory and/or cache. Portions of this software and/or data may be storedin any one of these storage devices. Further, the data and instructionscan be obtained from centralized servers or peer to peer networks.Different portions of the data and instructions can be obtained fromdifferent centralized servers and/or peer to peer networks at differenttimes and in different communication sessions or in a same communicationsession. The data and instructions can be obtained in entirety prior tothe execution of the applications. Alternatively, portions of the dataand instructions can be obtained dynamically, just in time, when neededfor execution. Thus, it is not required that the data and instructionsbe on a machine readable medium in entirety at a particular instance oftime.

Examples of computer-readable media include but are not limited torecordable and non-recordable type media such as volatile andnon-volatile memory devices, read only memory (ROM), random accessmemory (RAM), flash memory devices, floppy and other removable disks,magnetic disk storage media, optical storage media (e.g., Compact DiskRead-Only Memory (CD ROM), Digital Versatile Disks (DVDs), etc.), amongothers. The computer-readable media may store the instructions.

The instructions may also be embodied in digital and analogcommunication links for electrical, optical, acoustical or other formsof propagated signals, such as carrier waves, infrared signals, digitalsignals, etc. However, propagated signals, such as carrier waves,infrared signals, digital signals, etc. are not tangible machinereadable medium and are not configured to store instructions.

In general, a machine readable medium includes any mechanism thatprovides (i.e., stores and/or transmits) information in a formaccessible by a machine (e.g., a computer, network device, personaldigital assistant, manufacturing tool, any device with a set of one ormore processors, etc.).

In various embodiments, hardwired circuitry may be used in combinationwith software instructions to implement the techniques. Thus, thetechniques are neither limited to any specific combination of hardwarecircuitry and software nor to any particular source for the instructionsexecuted by the data processing system.

The description and drawings are illustrative and are not to beconstrued as limiting. The present disclosure is illustrative ofinventive features to enable a person skilled in the art to make and usethe techniques. Various features, as described herein, should be used incompliance with all current and future rules, laws and regulationsrelated to privacy, security, permission, consent, authorization, andothers. Numerous specific details are described to provide a thoroughunderstanding. However, in certain instances, well known or conventionaldetails are not described in order to avoid obscuring the description.References to one or an embodiment in the present disclosure are notnecessarily references to the same embodiment; and, such references meanat least one.

The use of headings herein is merely provided for ease of reference, andshall not be interpreted in any way to limit this disclosure or thefollowing claims.

Reference to “one embodiment” or “an embodiment” means that a particularfeature, structure, or characteristic described in connection with theembodiment is included in at least one embodiment of the disclosure. Theappearances of the phrase “in one embodiment” in various places in thespecification are not necessarily all referring to the same embodiment,and are not necessarily all referring to separate or alternativeembodiments mutually exclusive of other embodiments. Moreover, variousfeatures are described which may be exhibited by one embodiment and notby others. Similarly, various requirements are described which may berequirements for one embodiment but not other embodiments. Unlessexcluded by explicit description and/or apparent incompatibility, anycombination of various features described in this description is alsoincluded here. For example, the features described above in connectionwith “in one embodiment” or “in some embodiments” can be all optionallyincluded in one implementation, except where the dependency of certainfeatures on other features, as apparent from the description, may limitthe options of excluding selected features from the implementation, andincompatibility of certain features with other features, as apparentfrom the description, may limit the options of including selectedfeatures together in the implementation.

In the foregoing specification, the disclosure has been described withreference to specific exemplary embodiments thereof. It will be evidentthat various modifications may be made thereto without departing fromthe broader spirit and scope as set forth in the following claims. Thespecification and drawings are, accordingly, to be regarded in anillustrative sense rather than a restrictive sense.

What is claimed is:
 1. A system comprising: a server configured to:receive at least one thermal image of an environment, classify one ormore objects in the thermal image using an image processor, transmit anotification based on the classifying, adjust a setting of the imageprocessor in response to user feedback associated with the notification,and provide services based on the thermal image; a thermal imagingdevice mounted in the environment configured to capture the thermalimage; and a mobile device configured to: receive the notification,receive the user feedback regarding the notification via a userinterface, and transmit the user feedback to the server.
 2. The systemof claim 1, wherein the mobile device is configured to present anotification of a detected event in the environment and to receive userfeedback on the notification.
 3. The system of claim 1, wherein thethermal imaging device is configured to transmit the thermal image tothe server.
 4. The system of claim 1, wherein the mobile device isconfigured to transmit the thermal image to the server.
 5. The system ofclaim 1, wherein the mobile device is configured to pre-process thethermal image prior to transmitting the thermal images to the server. 6.The system of claim 1, wherein classifying the one or more objects inthe thermal image comprises identifying a human in the thermal image. 7.The system of claim 6, wherein the server further classifies a pose ofthe human.
 8. The system of claim 7, wherein the server further infersan activity in the image based on the pose and a floor plan of theenvironment.
 9. The system of claim 8, wherein the server generates thefloor plan based on one or more reference photographs captured by themobile device.
 10. The system of claim 1, the image processor comprisingan artificial neural network.
 11. A non-transitory computer-readablestorage medium for tangibly storing computer program instructionscapable of being executed by a computer processor, the computer programinstructions defining the steps of: receiving at least one thermal imageof an environment; classifying one or more objects in the thermal imageusing an image processor; transmitting a notification to a mobile devicebased on the classifying; receiving user feedback on the thermal imagefrom the mobile device; adjusting a setting of the image processor inresponse to the user feedback associated with the notification; andproviding services based on the thermal image.
 12. The computer-readablestorage medium of claim 11, wherein classifying the one or more objectsin the thermal image comprises identifying a human in the thermal image.13. The computer-readable storage medium of claim 12, wherein theclassifying further comprises classifying a pose of the human.
 14. Thecomputer-readable storage medium of claim 13, wherein the steps furthercomprise inferring an activity in the image based on the pose and afloor plan of the environment.
 15. The computer-readable storage mediumof claim 14, wherein the steps further comprise generating the floorplan based on one or more reference photographs captured by the mobiledevice.
 16. A method comprising: receiving, at a server, at least onethermal image of an environment; classifying, by the server, one or moreobjects in the thermal image using an image processor; transmitting, bythe server, a notification to a mobile device based on the classifying;receiving, at the server, user feedback on the thermal image from themobile device; adjusting, by the server, a setting of the imageprocessor in response to the user feedback associated with thenotification; and providing services based on the thermal image.
 17. Themethod of claim 16, wherein classifying the one or more objects in thethermal image comprises identifying a human in the thermal image. 18.The method of claim 17, wherein the classifying further comprisesclassifying a pose of the human.
 19. The method of claim 18, furthercomprising inferring an activity in the image based on the pose and afloor plan of the environment.